Simulation Guidée par l’Image pour la Réalité Augmentée durant la Chirurgie Hépatique

The main objective of this thesis is to provide surgeons with tools for pre and intra-operative decision support during minimally invasive hepaticsurgery. These interventions are usually based on laparoscopic techniques or, more recently, flexible endoscopy. During such operations, the surgeon tries to remove a significant number of liver tumors while preserving the functional role of the liver. This involves defining an optimal hepatectomy, i.e. ensuring that the volume of post-operative liver is at least at 55% of the original liver and the preserving at hepatic vasculature. Although intervention planning can now be considered on the basis of preoperative patient-specific, significant movements of the liver and its deformations during surgery data make this very difficult to use planning in practice. The work proposed in this thesis aims to provide augmented reality tools to be used in intra-operative conditions in order to visualize the position of tumors and hepatic vascular networks at any time.L’objectif principal de cette thèse est de fournir aux chirurgiens des outils d’aide à la décision pré et per-opératoire lors d’interventions minimalement invasives en chirurgie hépatique. Ces interventions reposent en général sur des techniques de laparoscopie ou plus récemment d’endoscopie flexible. Lors de telles interventions, le chirurgien cherche à retirer un nombre souvent important de tumeurs hépatiques, tout en préservant le rôle fonctionnel du foie. Cela implique de définir une hépatectomie optimale, c’est à dire garantissant un volume du foie post-opératoire d’au moins 55% du foie initial et préservant au mieux la vascularisation hépatique. Bien qu’une planification de l’intervention puisse actuellement s’envisager sur la base de données pré-opératoire spécifiques au patient, les mouvements importants du foie et ses déformations lors de l’intervention rendent cette planification très difficile à exploiter en pratique. Les travaux proposés dans cette thèse visent à fournir des outils de réalité augmentée utilisables en conditions per-opératoires et permettant de visualiser à chaque instant la position des tumeurs et réseaux vasculaires hépatiques

Calipso: Physics-based Image and Video Editing through CAD Model Proxies

We present Calipso, an interactive method for editing images and videos in a physically-coherent manner. Our main idea is to realize physics-based manipulations by running a full physics simulation on proxy geometries given by non-rigidly aligned CAD models. Running these simulations allows us to apply new, unseen forces to move or deform selected objects, change physical parameters such as mass or elasticity, or even add entire new objects that interact with the rest of the underlying scene. In Calipso, the user makes edits directly in 3D; these edits are processed by the simulation and then transfered to the target 2D content using shape-to-image correspondences in a photo-realistic rendering process. To align the CAD models, we introduce an efficient CAD-to-image alignment procedure that jointly minimizes for rigid and non-rigid alignment while preserving the high-level structure of the input shape. Moreover, the user can choose to exploit image flow to estimate scene motion, producing coherent physical behavior with ambient dynamics. We demonstrate Calipso's physics-based editing on a wide range of examples producing myriad physical behavior while preserving geometric and visual consistency.Comment: 11 page

Segmentation and Labelling of Intra-operative Laparoscopic Images using Structure from Point Cloud

International audienceWe present in this paper an automatic method for segmenting and labelling of liver its surrounding tissues in intra-operative laparoscopic images. The goal is to be able to distinguished between the different structure that compose a common intra-operative hepatic surgery scene. This will permits to improve the registration between pre-operative data and intra-operative images for task such as Augmented Reality. Our segmentation method consider the scene as a 3D structured point cloud instead of a laparoscopic images in order to exploit powerful informations such as curvature and normals, in addition to visual cues that permits to efficiently classify the scene. Our approach works well on sparse and noisy point clouds, thanks to a surface approximation stage, and unlike existing approaches, is independent of organs texture in the image. Experiements performed on challenging human hepatic surgery confirm that accurate segmentation and labelling are possible using 3D structure information and appropriate visual cues

Template-based Monocular 3D Recovery of Elastic Shapes using Lagrangian Multipliers

International audienceWe present in this paper an efficient template-based method for 3D recovery of elastic shapes from a fixed monocular camera. By exploiting the object's elasticity, in contrast to isometric methods that use inextensibility constraints , a large range of deformations can be handled. Our method is expressed as a saddle point problem using La-grangian multipliers resulting in a linear system which unifies both mechanical and optical constraints and integrates Dirichlet boundary conditions, whether they are fixed or free. We experimentally show that no prior knowledge on material properties is needed, which exhibit the generic usability of our method with elastic and inelastic objects with different kinds of materials. Comparisons with existing techniques are conducted on synthetic and real elastic objects with strains ranging from 25% to 130% resulting to low errors

Improving depth perception during surgical augmented reality

International audienceThis study suggests a method to compensate the loss of depth perception while enhancing organ vessels and tumors to surgeons. This method relies on a combination of contour rendering technique and adaptive alpha blending to effectively perceive the vessels and tumors depth. In addition, this technique is designed to achieve real-time to satisfy the requirements of clinical routines, and has been tested on real human surgery

Deformation-based Augmented Reality for Hepatic Surgery

International audienceIn this paper we introduce a method for augmenting the laparoscopic view during hepatic tumor resection. Using augmented reality techniques, vessels, tumors and cutting planes computed from pre-operative data can be overlaid onto the laparoscopic video. Compared to current techniques, which are limited to a rigid registration of the pre-operative liver anatomy with the intra-operative image, we propose a real-time, physics-based, non-rigid registration. The main strength of our approach is that the deformable model can also be used to regularize the data extracted from the computer vision algorithms. We show preliminary results on a video sequence which clearly highlights the interest of using physics-based model for elastic registration

Single View Augmentation of 3D Elastic Objects

International audienceThis paper proposes an efficient method to capture and augment highly elastic objects from a single view. 3D shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and resolve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or ressort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method which makes use of a me chanical model and is able to handle highly elastic objects. Our method is formulated as a energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. The only parameter involved in the method is the Young's modulus where we show in experiments that a rough estimate of its value is sufficient to obtain a good reconstruction. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of our approach. Experiments in the context of minimally invasive liver surgery are also provided

Réalité augmentée pour la chirurgie minimalement invasive du foie utilisant un modèle biomécanique guidé par l'image

National audienceCet article présente une méthode de réalité augmentée pour la chirurgie minimalement invasive du foie. Le réseau vasculaire et les tumeurs internes reconstruites à partir des données pré-opératoires (IRM ou CT) peuvent ainsi être visualisées dans l'image laparoscopique afin de guider les gestes du chirurgien pendant l'opération. Cette méthode est capable de propager les déformations 3D de la surface du foie à ses structures internes grâce à un modèle biomécanique sous-jacent qui prend en compte l'anisotropie et l'hétérogénéité du tissu hépatique. Des résultats sont montrés sur une vidéo in-vivo d'un foie humain acquise pendant une opération et sur un foie en silicone

Simultaneous Pose Estimation and Augmentation of Elastic Surfaces from a Moving Monocular Camera

International audienceWe present in this paper an original method to estimate the pose of a monocular camera while simultaneously modeling and capturing the elastic deformation of the object to be augmented. Our method tackles a challenging problem where ambiguities between rigid motion and non-rigid deformation are present. This issue represents a major lock for the establishment of an efficient surgical augmented reality where endoscopic camera moves and organs deform. Using an underlying physical model to estimate the low stressed regions our algorithm separates the rigid body motion from the elastic deformations using polar decomposition of the strain tensor. Following this decomposition, a constrained minimization, that encodes both the optical and the physical constraints, is resolved at each frame. Results on real and simulated data are exposed to show the effectiveness of our approach

Image-guided Simulation of Heterogeneous Tissue Deformation For Augmented Reality during Hepatic Surgery

International audienceThis paper presents a method for real-time augmentation of vas- cular network and tumors during minimally invasive liver surgery. Internal structures computed from pre-operative CT scans can be overlaid onto the laparoscopic view for surgery guidance. Com- pared to state-of-the-art methods, our method uses a real-time biomechanical model to compute a volumetric displacement field from partial three-dimensional liver surface motion. This permits to properly handle the motion of internal structures even in the case of anisotropic or heterogeneous tissues, as it is the case for the liver and many anatomical structures. Real-time augmentation results are presented on in vivo and ex vivo data and illustrate the benefits of such an approach for minimally invasive surgery